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Introduction 


Human activities such as deforestation, overfishing, and environmental 
pollution have endangered the survival of many species; for example, com- 
mercial whaling and marine garbage have brought whales close to extinction. 
To achieve a harmonious co-existence of humanity and nature, the United Na- 
tions has launched a program titled ‘Education for Sustainable Development’ 
Accordingly, understanding of ecological issues, such as climate change and 
biodiversity loss has become particularly stressed in environmental education 
and sustainable development (Mambrey et al., 2022), and the ecosystem has 
emerged as an important aspect in international science curricula (Australian 
Curriculum, Assessment and Reporting Authority, n.d.; Ministry of Education, 
&P.R. China, 2022; Next Generation Science Standards Lead States, 2013). 

Ecological issues refer to environmental problems concerning organ- 
isms, and their interactions with each other and the environment (Ecology 
Center, 2023). These are complex systems that are difficult to understand, 
because of the inherent intricacies, and long-term effects of human actions 
(Fanta et al., 2020). Multiple dimensions (e.g., social, economic, and socio- 
cultural elements) must be considered for identifying the consequences of 
such issues (Fanta et al., 2020; Ke et al., 2020; Riess & Mischo, 2010; Schuler 
et al., 2018). However, students lacking a systematic perspective cannot fully 
appreciate these issues (Ke et al., 2020; Liu et al., 2011; Yang, 2005). To solve 
these issues effectively, students need the systems thinking (ST) skills (Hogan, 
2002; Mambrey et al., 2022; Riess & Mischo, 2010) that are emphasized in 
international science education documents (e.g., National Research Council, 
2010; Next Generation Science Standards Lead States, 2013). ST is the skill to 
reason about complex systems in accordance with system characteristics to 
create a holistic understanding of various phenomena for problem-solving 
and system evaluation (Fanta et al., 2020; Gilissen et al., 2020; NRC, 2010). 
Researchers in various fields advocate the necessity to promote students’ ST 
in science learning (Gilissen et al., 2020; Schuler et al., 2018; Sommer & Licken, 
2010; Verhoeff et al., 2018). Moreover, complex and diverse perspectives of 
ecological issues would allow students to engage in ST about broader issues, 
and ultimately achieve global citizenship literacy (Ke et al., 2020; Lee et al., 
2013; Sadler et al., 2007). Thus, it is essential to explore and develop students’ 
ST in the context of ecological issues. 
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Abstract. Exploring students’ systems 
thinking (ST) is essential in enhancing 
science learning, but existing studies have 
failed to understand students’ ST fully as 
it relates to ecological issues. This study 
aimed to fill the aforementioned literature 
gap by exploring lower-secondary school 
students’ ST regarding ecological issues. 
The Systems Thinking Test regarding 
Ecological Issues, which measures four 

ST skills (system organisation, behaviour, 
application, and evaluation), was 
administered to 1,092 lower-secondary 
school students. The results reveal low ST 
performance in ecological issues, with 
students finding it particularly difficult to 
identify interactions among components 
and understand system characteristics. 
Furthermore, most lacked reflective 
consciousness and consideration of the 
diverse dimensions of ecological issues, 
resulting in monocausal reasoning in 
system decision-making and evaluation. 
Comparatively, urban school students 
performed better than their rural 
counterparts; additionally, an item-level 
analysis revealed that climate warming 
was challenging for the students to 
understand. This study suggests that 
greater efforts should be made to address 
students’ drawbacks and that multi- 
perspectival teaching is necessary in the 
context of ecological issues. The addition 
of system decision-making and evaluation 
in assessments can enable a broader 
understanding of ST. 
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ST in the Ecological Context 


According to previous studies, ST in science education incorporates not only the skill to identify interactions 
among system components and understand system characteristics (Ben-Zvi Assaraf & Orion, 2005; Brandstadter et 
al., 2012; Mehren et al., 2018; Sommer & Liicken, 2010) but also the skill to make predictions, decisions, and judge- 
ment based on system analysis (Ben-Zvi Assaraf & Orion, 2005; Evagorou et al., 2009; Mehren et al., 2018; Riess & 
Mischo, 2010) and make evaluations on the validity of system structures, decision-making, and the limitations of 
predictions (Fanta et al., 2020; National Research Council, 2010; Rosenkrdnzer et al., 2017). 

Given that multiple components and interactions of ecosystems must be considered, studies have mostly 
focused on students’ understanding of interdependent relations in ecosystems; for example, assessments of elemen- 
tary school students require them to reason about food webs (Hokayem & Gotwals, 2016; Mambrey et al., 2020; 
Mambrey et al., 2022). At the secondary level, they focus on students’ scientific explanations of ST concepts regard- 
ing ecosystems (e.g., indirect connections, feedback loops, and emergent properties) (Jin et al., 2019). However, 
these studies failed to cover the full range of ST skills needed by students for understanding and solving complex 
ecological issues, especially for making decisions and evaluating on the basis of system analysis. Although the 
direct impact of human actions on ecosystems has been incorporated in the assessment (Jin et al., 2019), it failed 
to provide opportunities for students to consider the other dimensions (e.g., society, economy, and socio-culture) 
indirect effects on causing or solving ecological problems. 


Research on Lower-secondary School Students’ ST Levels 


Most studies (e.g., Hokayem & Gotwals, 2016; Jin et al., 2019; Mehren et al., 2018) have assumed that there is 
a gradual increase in the development of students’ ST skills through learning progressions (LPs). Some researchers 
have defined ST using a three-level hierarchical model (from low to high), namely, analysis of system components, 
synthesis of system components, and problem-solving (Ben-Zvi Assaraf & Orion, 2010). Jin et al. (2019) defined 
a four-level LP for ST in an ecosystem based on students’ response complexity patterns; the four levels are called 
no ideas (Level 1), individual organisms (Level 2), relationships and patterns (Level 3), and mechanisms (Level 4). 
Mehren et al. (2018) differentiated systems-thinking levels according to system complexity, including the number 
of components and interrelations, type of relations, and dealing with system properties. 

Only a few ST measurements at the lower-secondary school level exist, especially for large groups of students 
(Brandstadter et al., 2012). Existing studies indicate a low starting point for ST among young students with regard 
to various fields. A small sample study of the hydro-cycle system indicated a relatively low performance of lower- 
secondary school students regarding ST skills in general (Ben-Zvi Assaraf & Orion, 2005). A small-scale comparison 
of the skill of understanding complex systems between experts and Grade 7 students using the Structure—Behav- 
iour-Function model implied that novice students focused on the structural level (Hmelo-Silver & Pfeffer, 2004). An 
assessment of ST regarding ecosystems has revealed that most secondary school students tend to identify direct 
and indirect relationships (Jin et al., 2019). However, the specific difficulties hindering lower-secondary school 
students’ ST in the context of ecological issues remain unclear. 

Studies have also revealed a great improvement in ST performance following the introduction of teaching in- 
terventions at the lower-secondary school educational level (Ben-Zvi Assaraf & Orion, 2005; Hmelo-Silver et al., 2017; 
Riess & Mischo, 2010); this indicates the feasibility and potential of developing lower-secondary school students’ ST. 


Students’ ST Performance in Different Groups 


For a comprehensive evaluation of ST, it is important to detect differences in students’ average performance 
among different subgroups. However, this question has not been sufficiently explored in ST research (Cox et al., 
2019; Jin et al., 2019). 

Gender science stereotypes have long been discussed; females are still considered to be underrepresented 
in different science fields (Brotman & Moore, 2008). However, the effect of gender on ST is unclear; research in dif- 
ferent scientific contexts has yielded distinct results. A study conducted in engineering found that male students 
performed significantly better than female ones (Sweeney & Sterman, 2000). However, research on ecosystems and 
geographical systems has suggested no significant differences based on gender (Cox et al., 2019; Jin et al., 2019). 
Mambrey et al. (2020) found a similar reasoning pattern in elementary and lower-secondary school students, sug- 
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gesting that the ST of young students regarding ecosystems was independent of their age. However, a significant 
difference was found with respect to geographical ST between grades, especially among high school students 
(Cox et al., 2019). Research has found significant performance gaps in ST among schools from different locations 
in the United States (Jin et al., 2019). Because the above-mentioned studies have shown distinct results in different 
system-specific contexts, it is necessary to check the effects of the aforementioned variables on students’ ST perfor- 
mance in the more complex context of ecological issues involving interactions between humans and ecosystems. 


Research Aim and Questions 


A full picture of students’ ST regarding ecological issues is essential for promoting ST to achieve sustainable 
development. However, the existing studies failed to cover the full range of ST skills and the specific weaknesses 
that hinder students’ development of ST. Moreover, it remains unclear in the literature whether students are able 
to contemplate the various dimensions involved in these complex issues. Thus, this study aimed to supplement 
the missing content by comprehensively evaluating on a large scale lower-secondary school students’ST regarding 
ecological issues. The following were the research questions for this study: 

1. What is the current level of ST regarding ecological issues among lower-secondary school students? 
2. Inwhich specific aspects of ST regarding ecological issues do students experience the most problems? 
3. How do students from different groups differ in their ST performance? 


Research Methodology 
Description of the Test to Measure Students’ ST 


The authors constructed a systems thinking test regarding ecological issues (STTEI) based on Mehren et al’s 
(2018) ST framework; it incorporated comprehensive ST skills based on the latest insights from ST studies and 
was empirically validated in the context of geography and ecology (Mambrey et al., 2020). The items of the STTEI 
measured four skills in ST, which we regard as covering the full range of skills included in the existing literature 
(Ben-Zvi Assaraf & Orion, 2005; Brandstadter et al., 2012; Evagorou et al., 2009; Fanta et al., 2020; Mehren et al., 
2018; National Research Council, 2010; Riess & Mischo, 2010; Sommer & Lucken, 2010), namely, system organisa- 
tion, system behaviour, system application, and system evaluation. Items in system organisation refer to identifying 
ecosystem components and the interrelations among components. Items related to system behaviour require 
students to analyse the results of system changes based on the interactions involved and their understanding of 
the system characteristics. To apply the system, students were asked to make predictions and decisions based on 
a system-effect analysis. For the last skill, students were required to work on problems and determine the validity 
of the system structure, decisions made, and limitations of predictions. Second, these ST skills were differentiated 
into three levels based on performance complexity according to system-related structural and behavioural com- 
plexity, as described by Mehren et al. (2018). These levels are described below. 


Level 1: Students identify low numbers of elements and isolated or monocausal relationships. Students’ 
analysis and reasoning regarding systems are based on monocausal relations. 


Level 2: Students identify a moderate number of elements and linear-indirect relationships. The analysis 
and reasoning are based on linear and indirect relationships. 


Level 3: Students identify a large number of elements and complex or nonlinear relations. Students per- 
form analysis and reasoning based on nonlinear and complex relations. 


The STTEI comprised 13 items (ordered multiple choice, short answers, two-tied items, and extended response 
questions) to probe different levels of students’ ST about ecological issues, and each item was allocated to one skill 
(Table 1). As climate change and sustainability problems have been the most prevalent ecological issues in the 
last decade (Fanta et al., 2020; You et al., 2018), the test was about issues highlighting sustainability as the context 
that could provide students with opportunities to explore multiple dimensions in ecological issues (e.g. climate 
warming, sustainable rainforest development, overfishing, industrial pollution, abuse of pesticides, and the absence 
of top predators). Items were scored using 0-2 or 0-3 point ranges according to the three levels of performance 
complexity. The two tied items and extended response questions gave students the opportunity to express their 
explanations of ecological issues. The concept map (CM) item is an example of an extended response question 
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(Figure 1). It was designed based on the context of climate warming to assess the extent to which students identify 
interrelations among components related to climate warming. The provided concepts included both biotic and 
abiotic components. The social element (industrial production) was included in this item. The other items of the 
STTEI are presented in the Appendix. 


Figure 1 
Concept Map Item (Item 5) 


Climate warming has influenced human life a lot. The following picture provides 
seven concepts interrelated with climate warming. Use the given concepts and think 
about how they relate to each other. Connect them with arrows accordingly and 
remember to label each arrow (see example). Make sure to indicate clearly the 
direction of the arrows. 


Carbon Dioxide 


Oxygen 


Animals and Human 


Industrial production 


Plants 


Petroleum, Coal 


and Mineral 


To ensure content validity, the item content and wording were evaluated and revised by expert review. A 
quantitative pilot test (n = 97) was conducted to validate the test instrument, and the Rasch partial credit model 
(PCM) was used to examine item fit statistics to address construct validity. Three items in the original version were 
eliminated because of low item discrimination (below .2). Subsequently, a field test (n = 456) was conducted to 
validate further the 13-item test. The expected a posteriori (EAP) reliability was .8, indicating acceptable internal 
consistency, and all items’ mean-squares fell into the suitable range of .7 to 1.3 (Bond & Fox, 2007). 


Table 1 
Specific Information of Test Items 


Skills required Item context Item number 
Identify system components Pond ecosystem Item 1 
System organisation 
Identify interrelations among components Climate warming Item 5 
Understand emergence Ecological equilibrium Item 3 
Dynamic change Absence of top predators Item 8 
System behaviour 
Identify feedback loop Temperature change in forests Item 11 
Cyclic thinking Nutrition cycle Item 10 
Make predictions Industrial pollutant, Item 2 
Climate warming Item 6 
System application 
Make decisions Climate warming, Item 7 
Sustainable development of rainforests Item 12 
Evaluate the system structure Overuse of pesticides Item 4 
System evaluation Determine the validity of the system structure, deci- Absence of top predators Item 9 
sions, and limitations of predictions Overfishing problems Item 13 
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Participants and Data Collection 


Asample representative of the diversity of lower-secondary school students is required to obtain more robust 
statistical results and meet the purpose of the study, such as the requirement of comparing students’ performance 
across genders, grades, and school locations. Thus, six lower-secondary schools in southern and northern regions 
of China were purposefully sampled. These schools varied in school settings, educational resources, and socio- 
economic status. According to the previous study, the sample size is determined by the standard error of measures, 
which should exceed the minimal requirement of 267 (Liu, 2020; Wu et al., 2016). In order to improve the accuracy 
of measurement, the sample size is four times larger than the required minimal sample size (the standard error 
of Rasch measures less than .1). All participants in these schools possessed knowledge about the ecosystem; a 
total of 1,183 students from Grades 7 and 8 involved in the study. After eliminating blank and invalid responses, 
information from 1,092 students was used for the analysis (Table 2). 

Data were collected from November 2022 to December 2022. The purpose of the study and test content were 
reported to school principals to acquire permission. All students and teachers participated in the study volun- 
tarily, and the test was conducted anonymously to protect students’ privacy. The teachers ensured that students 
completed the test within 45 minutes. Before the test, teachers informed students of the aim of the test and that 
the results would be used exclusively for scientific research purposes. The teachers also promised students that 
their test performance would not affect their subject grades and encouraged them to write down their honest 
thoughts as much as possible. 


Table 2 
Demographic Data of the Participants 


Demographic categories Number of students Percentage (%) 
Ss oAiinealion Urban schools 592 54.2 
Rural schools 500 45.7 
eee Male 565 51.7 
Female 527 48.2 
Gis Grade 7 568 52 
Grade 8 524 47.9 


Data Scoring and Analysis 


The developed scoring rubrics were used to score all students’ responses. Ten percent of the randomly selected 
tests were scored by two independent raters (a PhD student and a lower-secondary school biology teacher) to check 
inter-rater reliability. An inter-rater reliability analysis using kappa statistics determined the consistency between 
raters, and the kappa’s range was .62-.85, with a median of .75 (Cohen, 1968), indicating substantial agreement 
for each item from different raters (Landis & Koch, 1977). 

This study used PCM analysis via the Test Analysis Module package in R software (Robitzsch et al., 2021) to 
establish the participants’ ST levels on the test, as it has been deemed suitable for analysing polytomous data. The 
reliability of the EAP estimates was checked, and the weighted likelihood estimates (WLEs) were used as student 
proficiencies. A Wright (person-item) map was generated to quantify the locations of item difficulty and student 
performance on the same logit scale. The t-test was used to compare performances between the different sub- 
groups of participants (e.g., male vs. female). Differential item functioning (DIF) was applied to examine differences 
between student groups at the item level to provide further information on weaknesses and strengths with regard 
to students’ ST (Wu et al., 2016). 
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Research Results 
Students’ Overall Performance in ST Regarding Ecological Issues 


In Figure 2, the left side of the Wright map displays the density distribution (which approximates a normal 
distribution) of the student performance (WLEs) on the test. The right side represents the distribution of the 
Thurstonian thresholds for each item. A comparison of the WLEs and item thresholds showed that only a small 
proportion of students had more than a 50% chance of achieving the third level (Figure 2). 


Figure 2 
Wright Map for ST Regarding Ecological Issues 
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Items 
Note. 1 = threshold value 1, 2 = threshold value 2, and 3 = threshold value 3. 


The threshold values of each item level were calculated using PCM analysis (Table 3), and the mean threshold 
values of the item responses belonging to the same level across the ST skills were calculated (Wang & Song, 2021). 
The mean of the students’ ability measures (WLEs) was compared with the mean threshold values of the different 
levels; when the mean WLEs exceeded the latter, it was determined that the student had reached a specific ST 
level. As shown in Table 3, the student ability measures reached, on average, Level 1 for each skill, and Level 3 was 
the most difficult to achieve for each skill. The students’ mean ability (M = .047) for system organisation was higher 
than that for the other three skills. 
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Mean Threshold Values of Every Performance Level across ST 


; ; mean Participant 
Skill Level Items and threshold values on different levels Threshold ability 
values 
Levelt I1(-1.034); 15(-.341) -.688 
System organisation Level2 11(-.238); 15(1.253) 508 047 
Level3 11(1.156); 15(2.426) 1.791 
Levelt 13(-2.02); 18(-1.760) -.926 
System behaviour Level2 13(-.793); 18(-.594); 110(.313); 111 (-.232) 070 -.014 
Level3 13(3.092) 3.092 
Levelt 12(-2.141); 16(-2.294); I7(-1.791); 112(-.512) -1.685 
System application Level2 12(1.112); 16.402); 17(.359); 112(.633) 627 -.009 
Level3 12(3.099); 16(2.263); 17(2.554); 112(1.519) 2.359 
Levelt 14(-1.656); 19(-.592); 113(-.828) -.575 
System evaluation Level2 14(.794); 19(1.034); 113(.888) 1.619 -.004 
Level3 19(2.629); 113(2.525) 2.577 


Difficulties Experienced with Regard to Different ST Skills for Ecological Issues; 
Ability to Identify Components and System Interactions 


The analyses of the students’ identification of components in the pond ecosystem (Item 1) indicated that 
most students (57.5%) correctly identified a moderate (Level 2) or high (Level 3) number of elements (Figure 3). 
However, one-fifth of the students could not correctly identify the system components. The analyses of students’ 
wrong answers indicated that they had some misconceptions regarding hidden elements; for example, the students 
confused ‘green algae’ with ‘bacteria’ and sorted ‘bacteria’ into the producer and ‘green algae’ into the decomposer 
categories. Furthermore, some students misclassified ‘bacteria’ as an abiotic factor in the pond ecosystem. 


Figure 3 
Distribution of Students’ Responses on Items 1 and 5 
Item 1 Item 5 
Level 3, Level 3, 
as 20.5 . soo 
2 Level 2, 37g Level 2, ae 
2 n=4-5 ~ 3<SX<.6 ; 
o n 
2 Level 1, 5 Level 1, 
2 13 = RF 1<SX< 3 = 
8 x 
~ Level 0, Level 0, 
13 20.3 SX< 1 40.3 
0% 10% 20% 30% 40% 0% 10% 20% 30% 40% 50% 
Proportion of students Proportion of students 


Note. n = number of components. The structural index (SX) = sum of all arrow strings, branches, and cycles / the number of 
concepts (Mehren et al., 2015). 
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Although the students performed well in identifying the system components, analysis of the CMs indicated 
that most students possessed an incomplete picture of climate warming. Students’ abilities to identify interrelations 
among components were assessed using the structural index (SX) of the CM (see Figure 3). The results of the CMs 
showed that most students’ (82%) SX scores in Item 5 were less than .3 (Figure 3), indicating students’ low ability to 
identify interrelations among system components. Figure 4 shows three typical networks that visualise the struc- 
ture of the students’ CMs at the three levels. The first network in Figure 4 indicates that students at Level 1 could 
not make connections among all concepts provided in the CM and tended to identify direct linear connections 
among concepts; 14.2% of students reached Level 2, and students at this level could make indirect relations among 
concepts (e.g. branches [at least two arrows lead to/away from a concept]) and identify a simple carbon cycle via 
plant photosynthesis and respiration of organisms in the second network of Figure 4. Only a few students (3.5%) 
connected all the concepts in a networked manner. Complex relationships (e.g., different cycles and interactions 
among oxygen, plants, and carbon dioxide) can be identified at Level 3 (Figures 3 and 4). 


Figure 4 
Examples of the Networks in Students’ Concept Maps at the Three Levels 
Level1 students’ concept map Level2 students’ concept map 
Carbon Dioxide Industrial production 
Plants Mineral fules 


Animals & Human 


Oxygen Carbon Dioxide 


Pants 
Animals & Human Oxygén 


Level3 students’ concept map 


Industrial preduction 
Oxygen 


Animals & Human 
Carbor Dioxide 


Plants 


Mineral fules 
Decomposer 


Ability to Analyse System Behaviour and System Characteristics 


The results of Item 3 suggest that most students (54.1%) recognised dynamic changes among the three 
populations (wolves, deer, and plants), which help to maintain the equilibrium of the ecosystem. Furthermore, 
students’ responses to Item 8 also support the result that most students (64.6%) were able to explain the disap- 
pearance of grey wolves, which caused an increase in the number of deer, which, in turn, damaged plants. However, 
students’ inability to use emergent properties (e.g., carrying capacity and energy pyramid) to explain the changes 
in the ecosystem suggests that incomprehension of system characteristics hindered students’ analysis of dynamic 
behaviours in the system. Item 3 showed that only a few students (4.4%) could explain the long-term changes in 
the number of animals based on the conception of the energy pyramid (Table 4). 

Furthermore, the t-test analysis of Items 10 (M = .790) and 11 (M = 1.170) revealed that students performed 
significantly better in recognising the feedback loop than the nutrition cycle in the forest (¢ (2,147.7) = -8.92, p < 
.001), and only 39% of students could correctly connect the process of the nutrition cycle in Item 10. These find- 
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ings indicate that a lack of cyclic thinking makes it difficult for students to combine a variety of relationships into 
coherent cyclic and dynamic processes. Analysis of the CM also revealed a similar result; the third network in Figure 
4 suggests that different carbon cycles exist in the context of climate warming (e.g., interactions via photosynthesis 
and the respiration of plants, cyclical processes such as plants’ photosynthesis and the respiration of organisms, and 
the connection of mineral fuels to industrial emissions or human consumption). However, students at Levels 1 and 
2 (56.2%) only formed a fragmented perception of the carbon cycle in the context of climate warming (Figure 4). 


Table 4 
Item 3 System Characteristics and Contents 


System characteristics Specific contents Number of students 
Direct/indirect interactions The relations in food chains 254 (22.9%) 
Dynamics and stability Dynamic changes among populations 599 (54.1%) 
Emergent property Energy pyramid 49 (4.4%) 


Ability to Make Predictions and Decisions 


Figure 5 displays the proportion of student responses at different ST levels. The size of each chunk in the 
bar represents the proportion of the students. Figure 5 suggests that a large proportion of students’ responses 
were at Level 1, which implies that monocausal and direct thinking dominated the students’ systems application. 
Furthermore, students’ preference to identify direct effects closely related to the environment also supports this 
result; see, for instance, the common responses in Item 2: ‘Pollution of air and water caused the deaths of the 
organisms in the water’ Only a small proportion of students who formed a complex analysis at Level 3 (Figure 5) 
could anticipate the long-term effects based on a comprehensive understanding of the ecological, economic, and 
sociocultural dimensions of ecological issues (e.g., effects of industrial pollutants on residents’ income in Item 2 
and decision-making considering the sustainable development of the rainforest as well as the protection of local 
people's lifestyle in Item 12). 


Figure 5 
Students’ Response Distribution Regarding the Items in System Application and System Evaluation 


0, 
sca 6.70%: 
80% 
60% 

: CG Level 3 

40% Level 2 

u (I Level 1 

i Level 0 
20% 
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Item 6 Item 7 Item 12 Item 9 Item 13 


System Application System Evaluation 


Note. Level 0 represents incorrect responses; Level 1 represents monocausal relations; Level 2 represents indirect/linear relations, 
and Level 3 represents complex relations. 
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Ability to Make Evaluations about the Structures and Applications of Systems 


The results of Item 4 showed that holistic thinking was lacking in students’ evaluation in that most students 
(84.3%) could find the error of isolated relations in the system, but few students (31.1%) could evaluate the relations 
from a system perspective. To make the right judgement about relation 9 (‘human consumption of crop’ is good 
for‘human health’) in Item 4, students had to think about the whole system instead of just the parts, because the 
overuse of pesticides would result in crops accumulating pesticide residues. However, more than half of the stu- 
dents (53.2%) overlooked this whole system background. Furthermore, the results for Items 9 and 13 suggest that 
most students made evaluations in a monocausal (Level 1)/incorrect (Level 0) manner (Figure 5). Further analysis 
of students’ responses indicated that these students lacked reflective consciousness about the limitations of pre- 
dictions and decisions and could not make full evaluations of the validity of decisions and predictions in a time 
dimension; for example, they were unable to determine the long-term effects of killing grey wolves on ecosystems 
and that of a fishing ban on the economy and society. 


Identification of Performance Gaps in Different Student Groups 


The t-test was used to compare the means of student performances using WLEs for particular student 
groups (Figure 6). Students showed significantly different levels of performance based on their school location 
(t (1,090) = -8.388, p < .001); rural students (M = -.236) showed lower performance than urban students (M = .187). 
The results suggested that there was no significant difference between male (M = .033) and female students 
(M = -.049; t (1,090) = 1.583, p = .114), and no significant difference between Grades 7 (M = .01) and 8 (M = -.02) 
students (t (1,090) = .672, p = .502). 

In the DIF analysis, Items 2, 5, 6, 7, and 9 appeared to be more difficult for rural school students, at a .01 
significance level. The specific content and ST skills regarding these items were examined, and the analysis of stu- 
dents’ responses indicated that rural students had more difficulties in identifying interactions among components 
and making predictions, decisions, and evaluations in an indirect or complex way than their urban counterparts. 
Furthermore, Items 5, 6, and 7 were all included under the context of climate warming, which means that this 
ecological issue is challenging for rural school students to understand. 


Figure 6 
Differences in ST Performance between Various Student Subgroups 
<a 251. 2.54 : 
2.54 e bd 
ol °om)|,oe 
7) 7) 7) 
LW Ls Ww Lu 
—! _ _I 
= = = 
95, ° : 25-0 | : 25) | : 
Rural Urban Grade 7 Grade 8 “Female Male 
School Setting Grade Gender 


Note. *** <.001. 


874 


ASS https://doi.org/10.33225/jbse/23.22.865 


Journal of Baltic Science Education, Vol. 22, No. 5, 2023 


ISSN 1648-3898 /Print/ EXPLORING LOWER-SECONDARY SCHOOL STUDENTS’ SYSTEMS THINKING PERFORMANCE 

IN ECOLOGICAL ISSUES 

ISSN 2538-7138 /Online/ (PP. 865-880) 
Discussion 


This study incorporates a broader set of ST skills (e.g., system decision-making and evaluation) for assessing stu- 
dents’ consideration of diverse dimensions of ecological issues, resulting in a more comprehensive understanding of 
their ST performance in ecological issues. Furthermore, this study recruited a larger sample group than previous studies, 
thereby yielding more reliable and representative baseline data on lower-secondary school students’ performance. 

Regarding RQ1, the Wright map and the comparison of students’ abilities both suggest that, on average, the 
students reached Level 1 of ST about ecological issues. The low ST performance of the participants is not surprising 
because some previous studies from different contexts also found similar results (Ben-Zvi Assaraf & Orion, 2005; 
Mambrey et al., 2020). Among the four skills in ST, systems organisation was found to be relatively easier for students, 
which is consistent with the findings of Evagorou et al. (2009) and Hmelo-Silver and Pfeffer (2004), who observed 
that the structure of a system was the most cognitively easy for students to understand. 

The analysis of the students’ responses to the items in RQ2 further explained the reasons for their low perfor- 
mance in understanding ecological issues. Students exhibited difficulties comprehending the interrelations among 
system components and identifying hidden elements in the ecosystem. The microorganism is invisible and thus easily 
neglected, making it difficult for students to identify them (Ben-Zvi Assaraf & Orion, 2005). Results in this study align 
with previous research findings that emergence is the most difficult system characteristic for students to understand 
(Hmelo-Silver et al., 2007; Jin et al., 2019; Sommer & Luicken, 2010). However, the analysis implies that stability and 
dynamics are relatively easy for the lower-secondary school students in China. The ecosystem equilibrium highlighted 
in the Chinese lower-secondary school biology curriculum likely promotes students’ understanding of the character- 
istics. The little to no use of system concepts in explanations also supports the necessity of imparting general system 
characteristics (e.g., feedback loop, emergent properties, equilibrium) to scaffold students’ coherent understanding 
of complex systems from a system perspective (Gilissen et al., 2020; Gilissen et al., 2021). Students lacked the cyclic 
perception, as reported by Ben-Zvi Assaraf et al. (2005) for the water cycle, and most students were unable to draw 
a complete network of interactions. The lack of time dimension hinders students’ ability to make predictions and 
decisions based on long-term effects. 

Although evaluation is an essential aspect of ST (Fanta et al., 2020; National Research Council, 2010), it is usually 
ignored in assessment. This study's findings indicate that students were less proficient in system decision-making 
and evaluation. There may be several reasons for this. First, the absence of reflective consciousness made it difficult 
for students to contemplate the limitation of prediction and decision. Second, the lack of holistic thinking prevented 
most of them from deciding or evaluating from a comprehensive perspective. The results also suggest students’ in- 
ability to recognise other dimensions (such as social, economic, and cultural), and the indirect effects on ecological 
problems in their predictions, decision-making and evaluation. This further explains the dominance of monocausal 
reasoning in most students’ ST, and their inability to fully understand the causes of ecological problems. This led them 
to decide and evaluate from a unidimensional perspective. 

Regarding RQ3, this study's results contradict those of Jin et al. (2019), who concluded that the average per- 
formance of urban students was statistically significantly lower than that of rural students. A major reason for this is 
the contrast in urban area educational resources between China and the United States. Urban schools in the United 
States tend to have low resources, high teacher turnover, and low-quality students (Jin et al., 2019). However, with 
the accelerating urbanisation in China, resources have accumulated in urban areas. In general, urban schools in China 
have better educational resources and more highly qualified teachers than rural schools (Wen & Gu, 2017). Given that 
context-specific knowledge is necessary for developing ST (Ben-Zvi Assaraf & Orion, 2005; Sommer & Lucken, 2010), 
the lower performance of rural students in the context of global warming implies a possible lack of knowledge about 
this ecological issue. To explain the differences between rural and urban education more precisely, future research 
should explore the extent to which possible variables (e.g., socioeconomic status, teachers’ teaching quality, and 
students’ knowledge about ecological issues) influence students’ ST. 


Conclusions and Implications 
This study explored lower-secondary school students’ system thinking performance in ecological issues. The 
STTEI was administered for acquiring extensive data from Chinese students. Their overall ST performance was at 


Level 1, and an analysis of their responses revealed that they faced substantial difficulties in understanding ecologi- 
cal issues. Furthermore, this study is unique in revealing that most lower-secondary school students lack reflective 
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consciousness, and are thus unable to consider the indirect impact of diverse dimensions, leading to monocausal 
reasoning in system application and evaluation. This study also found that Chinese rural students did not perform 
as well as their urban counterparts, specifically in the context of climate warming, which may be explained by rural 
school students’ lack of the corresponding content knowledge. 

These findings have significant implications, particularly for developing lower-secondary students’ ST in 
ecological issues. First, teachers must address students’ shortcomings, especially regarding emergent properties, 
cyclic perception, and interactions within complex systems. Second, ecological issues must be discussed from 
multiple perspectives for promoting ST in science classes and improving students’ causal reasoning in complex 
systems. Third, there should be greater efforts in making rural students understand climate change. Future research 
must examine other variables for developing teaching strategies suited to students’ backgrounds. Furthermore, 
qualitative methods (e.g., semi-interview) should be used for future assessments, so that the underlying causes of 
students’ low performance are identified. 
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Appendix 


Sunlight 


Item 1 Answer the following questions based on the information 
provided in the picture: 


Organisms that use the energy of sunlight to produce organic matter 
include 


Abiotic factors affecting the pond ecosystem include 


@ am 
SSS 


os 
Large Fish 


small He 


The decomposer in the pond is 


Magnified View 
of Green Algae 


seemed ics (Modified from the National Assessment of Educational Progress, 
cuore http://nces.ed.gov/NATIONSREPORTCARD/) 


Item 2 A newly built factory near the pond emits substantial amounts of sulphur dioxide and other pollutants and 
discharges wastewater containing heavy metal pollutants, nitrogen, and phosphorus (nitrogen and phosphorus 
will cause many green algae to multiply) into the pond. What do you think this will do to the creatures in the pond 
and the surrounding villagers? 


Item 3 A remote island is uninhabited. The main animals on the island are wolves and deer, and the island is free 
from external interference all year round. 


(1) Select the answer below that best describes what will happen to the average populations of the animals over 
time. ¢ ) 
a) Thedeer will all die or be killed. 
b) The wolves will all die. 
c) There will be many more deer than wolves. 
d) There will be many more wolves than deer. 
e) The populations of each will be approximately equal. 
(2) Choose the reason for your answer ( ) 
a) Wolves prey on deer, and deer are eaten up by wolves. 
b) Wolves prey on deer, causing the number of deer to decrease. 
c) Wolves prey on deer. The number of deer decreases. Wolves die as they lack food. 
d) Tomaintain the ecological balance of the ‘plant > deer > wolf’food chain, the number of wolves and 
deer will fluctuate up and down within a certain range. 
e) When energy is transmitted along the food chain, it decreases step by step. The population of wolves 
store less energy than the population of deer. 


Item 4 The following concept map describes the impact of heavy use of pesticides in farmland on people and the 
ecosystem. There are two errors in the word relationships described in the concept map. Write the wrong serial 
number and explain the correct relationship. 


large use 
of 
pesticide 


10.result in 


l.result in 


water 
pollution meee le 
P 3 ~ [zeduction 3.result in 
— 


14! ss 2. promote 
harmful to the number of 
growth of natural enemies of 
crops pests decreases 
human \ v4 
health 5. increase 6. promote 4. in favour 
4 of 
crop 
yield BS breed of 
9.in favour or 8. promote 


of 


7. increase 
economic 
human growth 
consumption 
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Item 6 Together with the concept map you have constructed, please predict what will happen if the number of 
plants substantially decreases. 


Item 7 Everyone is responsible for protecting the Earth as our home. Please put forward specific measures to miti- 
gate climate warming. 


Please answer the following questions according to the information in the food web (the arrow in the food web 
points to the predator). 


Item 8 lf all grey wolves disappear, what will be the trend in the 
deer population ( ) 


grey 
wolves 


eagles 


A. It will keep growing. C. It will keep decreasing. 


B. It will grow first, then decrease. _ D. It will decrease first, then 
grow again. 


deer 


Explain the reasons for your answer: 


insects 


polar 


Item 9To protect the safety of villagers near the forest, some people proposed to kill all the grey wolves in the for- 
est, but Ming argued that ‘if there are no grey wolves in the forest, the number of all herbivores in it will increase 
rapidly in a short period, thus damaging the stability of the ecosystem’ Do you think Ming's analysis is correct? 
Please explain your reasons. 


Item 10 Use arrows to connect the following words to form the nutrient cycle of poplar trees in the forest. The 
first arrow has been provided in the picture, and the last arrow you draw must point to ‘poplar trees’ 


Falling of leaves 


Absorption of nutrients 


via the roots 


Release of nutrients 


Decomposition by fungi 


and bacteria in the soil 


Plant growth from 
nutrients 


Poplar trees 


(Modified from Mehren et al., 2018) 
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Item 11 Trees and plants can reduce ambient temperature through transpiration. The higher the ambient 
temperature, the stronger the transpiration of plants, and the more water in plants is emitted into the air. Please 
judge what changes will happen to the forest environment when the temperature rises and the transpiration of 
plants increases. Fill in ‘promote’ or‘weaken at A, B, and C in the following figure (as shown in the example below). 
Please fill in the blanks in the order of A> B > C. 


Item 12 The Khmus people in Xishuangbanna have been using slash-and-burn cultivation in the tropical 
rainforest. They cut down and burn trees to clear the land for planting food. The ashes from the burning of trees 
can increase soil fertility. This planting method is only effective for one year, and the land fertility will decline in the 
next year. The rainforest includes valuable trees (with high prices) and ordinary trees. After cutting them down, it 
takes at least ten years to recover the trees. 


(1) To achieve sustainable development of tropical rain forests and the planting activities of Khmus, which of the 
following measures do you think is the most reasonable? 


A. Cut down all the trees in the rainforest and sell the valuable trees. 
B. Only cut down the valuable trees in the rainforest and sell all the valuable trees. 


C. Divide the entire rainforest area into ten equal segments. Every year, cut down trees only in a partial 
area and sell the resulting valuable wood. 


D. Divide the entire rainforest area into ten equal segments. Every year, cut down trees only in a partial 
area and sell all the resulting wood. 


(2) Explain the reasons for your answer. 


Item 13 Zhoushan in Zhejiang Province is known as the ‘Eastern Fish Warehouse’ in China. People here basically 
rely on fishing to make a living. However, with the overfishing of yellow croakers, the number of yellow croakers 
has declined sharply in the past few years. One reason is that many young yellow croakers have been caught. To 
prevent the extinction of wild yellow croakers, the measure of ‘banning fishing within three years’ was proposed. 
Do you think this measure is feasible? Please explain your reasons. 
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